Add DXVK-native (D3D9 on Vulkan) as an alternative to ToGL on Linux - #492

This adds an optional Vulkan renderer on Linux via DXVK-native, one of the items under "Current tasks" in the README.

With ./waf configure --use-dxvk the engine compiles its real Direct3D 9 path (the one used on Windows) against DXVK's headers instead of ToGL. DXVK then translates D3D9, including the shipped D3D9 shader bytecode, to Vulkan. Without --use-dxvk nothing changes: ToGL stays the default, and the Windows, macOS and Android builds are untouched.

How to build

scripts/build-ubuntu-amd64-dxvk.sh

This works like build-ubuntu-amd64.sh, but first checks out the dxvk submodule and builds DXVK (d3d9 only, SDL2 WSI) with meson into dxvk-x86_64/. It then configures the engine with --use-dxvk. To run the game, copy dxvk-x86_64/lib/libdxvk_d3d9.so* next to the engine libraries in bin/.

If you build by hand, use ./waf configure --use-dxvk [--dxvk-prefix=<dir>]. The prefix defaults to dxvk-<arch>/ and must contain include/dxvk and lib/libdxvk_d3d9.so.

The DXVK submodule

• dxvk/ is doitsujin/dxvk pinned at v3.1.1.
• It has update = none in .gitmodules, so git submodule update in the existing scripts skips it. Only the DXVK script checks it out.
• It has ignore = untracked, because newer meson writes subprojects/.wraplock into the source tree.

I noticed there is also nillerusr/dxvk-native. That is Joshua Ashton's DXVK-Native at 1.9.2a (2021). Native support has been part of upstream DXVK since 2.0, so I used upstream v3.1.1. If you prefer your own fork as the submodule, it would need to be updated to a 2.x/3.x release, and switching the URL is easy.

Engine changes

• public/togl/dxvk_d3d9.h (new): includes DXVK's windows.h, d3d9.h, d3dx9math.h and d3dx9shader.h. DXVK's ULONG (32-bit) and HMODULE are renamed while including, because the engine's POSIX typedefs differ. ID3DXBuffer is declared by hand, because the full d3dx9.h needs GDI types that DXVK does not have.
• shaderapidx9/dxvk_d3dx9.cpp (new): the D3DX9 helpers the shader API calls, because there is no d3dx9 library on Linux. These are matrix math, ID3DXMatrixStack and D3DXGetShaderVersion. D3DXCompileShader returns E_NOTIMPL. GetVidMemBytes() returns 2 GB, because D3D9 cannot tell.
• winutils: GetClientRect, IsIconic and ClientToScreen are implemented with SDL (HWND is the SDL_Window*). Sleep, GetCurrentThread and SetThreadAffinityMask are shared with ToGL. The DebuggerBreak() stays only for ToGL, because the D3D9 path really calls these functions.
• sdlmgr: the window is created with SDL_WINDOW_VULKAN and no GL context.
• sys_mainwind: the D3D9 device window is the SDL window.
• shaderdevicedx8: sets DXVK_WSI_DRIVER=SDL2 unless it is already set. DXVK-native throws an error without it.
• platform.h: IsPlatformOpenGL() is false with DXVK, so the Windows rendering paths are used (SM3 shaders, D3D9 vendor workarounds).
• imageformat.h: on POSIX this defines a fake D3DFORMAT enum with ToGL values. With DXVK the real enum is used, and the D3DFORMAT helpers are declared only where dxvk_d3d9.h is included. This keeps the Windows headers out of the rest of the engine.
• materialsystem, shaderapidx9 wscripts: the strncpy and _snprintf poison defines are dropped for DXVK builds. They break <cstring> from DXVK's headers (using ::strncpy).
• dxabstract: the AddRef/Release debug arguments now depend on #if defined( DX_TO_GL_ABSTRACTION ) instead of #if POSIX.
• wscript: the --use-dxvk and --dxvk-prefix options. togl is not built with DXVK.

Testing

• x86_64 laptop (Fedora 44, Intel Arc B390, Mesa 26.2, DXVK v3.1.1), timedemo on a d1_canals_01 demo (5449 frames, 1280×720), release builds:
  • ToGL: 725 fps
  • DXVK: 1004–1347 fps
• riscv64 (Milk-V Jupiter, Radeon R9 290 via RADV): 57–59 fps with DXVK vs 51–52 fps with ToGL. RISC-V support itself will be a separate PR.
• Rendering: the main menu (water reflections, fonts) and d1_canals_01 render the same as with ToGL.
• Build check: the default ToGL build still compiles with these changes. scripts/build-ubuntu-amd64-dxvk.sh builds from a fresh clone.

I did not add a CI job. The existing amd64 job runs on Ubuntu 20.04, whose meson and gcc are probably too old for DXVK 3.x. A DXVK job would have to run on ubuntu-latest. I can add one if you want it.

How this was made

I made this together with Claude Code, Anthropic's AI coding assistant. I guided it throughout and corrected it where needed, so that the change fits the existing project structure:

• the same kind of build script as the other scripts/build-ubuntu-*.sh
• a submodule like the existing ones, with no extra container or build tooling
• shared code instead of duplicated code where possible (for example in winutils)
• ToGL behaviour left exactly as it was

I tested the result myself on both machines listed above.